Skip to content

[google_workspace] drop events that have missing items[] in the raw logs from the API #15948

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[google_workspace] drop events that have missing items[] in the raw logs from the API #15948

wants to merge 2 commits into from
+341 −282

Conversation

@brijesh-elastic
Copy link
Collaborator

@brijesh-elastic brijesh-elastic commented Nov 12, 2025 

google_workspace: Discard events that are missing the `items[]` field during the split operation
and returned as the root object

The Google Workspace Reports API sometimes does not return the `items[]` array, resulting
in the absence of the target field in the `response.split` operation. This leads to the
root level object being returned, which causes failures in the ingest pipeline.

To address this issue, a `drop` processor has been added at the start of the pipeline to ensure
that we discard events that are not required.

Here is the list of affected data streams:

- access_transparency
- admin
- context_aware_access
- device
- drive
- gcp
- group_enterprise
- groups
- login
- rules
- saml
- token
- user_accounts

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/google_workspace directory.
  • Run the following command to run tests.

elastic-package test -v

@brijesh-elastic
Discard events that are missing the target during the split operation…
1df316d 
… and are subsequently returned as the root object
@brijesh-elastic brijesh-elastic self-assigned this Nov 12, 2025
@brijesh-elastic brijesh-elastic requested a review from a team as a code owner November 12, 2025 10:03
@brijesh-elastic brijesh-elastic added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:google_workspace Google Workspace bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Nov 12, 2025
@elasticmachine
Copy link

elasticmachine commented Nov 12, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@elasticmachine
Copy link

elasticmachine commented Nov 12, 2025

💚 Build Succeeded

cc @brijesh-elastic

chrisberkhout
chrisberkhout reviewed Nov 12, 2025
View reviewed changes
Copy link
Contributor

@chrisberkhout chrisberkhout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the problem, at least as described, should be avoided by using the ignore_empty_value option.

If that's not the case, can you please explain why not?

A system test could exercise this, but I think what's important is to first just understand what is going on.

@brijesh-elastic
Copy link
Collaborator Author

brijesh-elastic commented Nov 12, 2025

I think the problem, at least as described, should be avoided by using the ignore_empty_value option.
If that's not the case, can you please explain why not?

We already have ignore_empty_value: true set (see here), but events are still being published.
If we have one level of response.split, then the event will be discarded when ignore_empty_value: true is configured.

@chrisberkhout
Copy link
Contributor

chrisberkhout commented Nov 13, 2025

If we have one level of response.split, then the event will be discarded when ignore_empty_value: true is configured.

Okay, so it's not working because it's multi-level. That sounds like a beats bug.

I'll take a look there and probably file an issue. If so, we can merge this workaround with a comment that refers to the issue and we should probably also look for the same issue in other integrations.

@brijesh-elastic
Copy link
Collaborator Author

brijesh-elastic commented Nov 14, 2025

@chrisberkhout, Sounds good. Please share the issue link once you confirm and file it so that I can add a comment with the link to the ingest pipeline. Meanwhile, I'll look for other integrations that may have the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chrisberkhout chrisberkhout Awaiting requested review from chrisberkhout

At least 1 approving review is required to merge this pull request.

Assignees

@brijesh-elastic brijesh-elastic

Labels

bugfix Pull request that fixes a bug issue documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:google_workspace Google Workspace Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@brijesh-elastic @elasticmachine @chrisberkhout